Methodological quality of systematic reviews in subfertility: a comparison of Cochrane and non-Cochrane systematic reviews in assisted reproductive technologies.

STUDY QUESTION Are there differences in the methodological quality of Cochrane systematic reviews (CRs) and non-Cochrane systematic reviews (NCRs) of assisted reproductive technologies? SUMMARY ANSWER CRs on assisted reproduction are of higher methodological quality than similar reviews published in other journals. WHAT IS KNOWN ALREADY The quality of systematic reviews varies. STUDY DESIGN, SIZE AND DURATION This was a cross-sectional study of 30 CR and 30 NCR systematic reviews that were randomly selected from the eligible reviews identified from a literature search for the years 2007-2011. MATERIALS, SETTING AND METHODS We extracted data on the reporting and methodological characteristics of the included systematic reviews. We assessed the methodological quality of the reviews using the 11-domain Measurement Tool to Assess the Methodological Quality of Systematic Reviews (AMSTAR) tool and subsequently compared CR and NCR systematic reviews. MAIN RESULTS AND THE ROLE OF CHANCE The AMSTAR quality assessment found that CRs were superior to NCRs. For 10 of 11 AMSTAR domains, the requirements were met in >50% of CRs, but only 4 of 11 domains showed requirements being met in >50% of NCRs. The strengths of CRs are the a priori study design, comprehensive literature search, explicit lists of included and excluded studies and assessments of internal validity. Significant failings in the CRs were found in duplicate study selection and data extraction (67% meeting requirements), assessment for publication bias (53% meeting requirements) and reporting of conflicts of interest (47% meeting requirements). NCRs were more likely to contain methodological weaknesses as the majority of the domains showed <40% of reviews meeting requirements, e.g. a priori study design (17%), duplicate study selection and data extraction (17%), assessment of study quality (27%), study quality in the formulation of conclusions (23%) and reporting of conflict of interests (10%). LIMITATIONS, REASONS FOR CAUTION The AMSTAR assessment can only judge what is reported by authors. Although two of the five authors are involved in the production of CRs, the risk of bias was reduced by not involving these authors in the assessment of the systematic review quality. WIDER IMPLICATIONS OF THE FINDINGS Not all systematic reviews are equal. The reader needs to consider the quality of the systematic review when they consider the results and the conclusions of a systematic review. STUDY FUNDING/COMPETING INTEREST(S) There are no conflicts with any commercial organization. Funding was provided for the students by the summer studentship programme of the Faculty of Medical and Health Sciences of the University of Auckland.